What is claimed is: 

1. A method for synthesizing moving images, comprising 
the steps of: 

sampling two consecutive frames from a moving image; 
5 providing a reference patch having at least one 

rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
reference patch, in the other of the two frames; 
10 moving and/or deforming the second patch in the other of 

the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
15 the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
20 space of the reference frame, based on the correspondent 
relationships; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame; 
25 varying the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner, 
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estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
thereby obtaining a plurality of correlative values 
5 corresponding to the number of rectangular regions within the 
reference patch and the second patch; 

comparing the degrees of correlation for each number of 
rectangular regions, based on the plurality of the correlative 
values; and 

10 ~ generating a synthesized frame having a higher resolution 
than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
reference patch, based on the correspondent relationships, 
which were estimated among the pixels of the reference patch 

15 and the second patch having the number of rectangular regions 
therein that yielded the highest degree of correlation. 

2. A method for synthesizing moving images, comprising 
the steps of: 

sampling at least three consecutive frames from a moving 

2 0 image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

25 providing a second patch, which is the same as the 

reference patch, in another of the plurality of sampled frames; 
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moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
within the reference image; 

estimating correspondent relationships among the pixels 
5 within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
10 converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 
15 patch of the reference frame and the coordinate converted frame; 

varying the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
20 correlative values for each number of rectangular regions, 
thereby obtaining a plurality of correlative values 
corresponding to the number of rectangular regions within the 
reference patch and the second patch; 

comparing the degrees of correlation for each number of 
25 rectangular regions, based on the plurality of the correlative 
values; 



155 



obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 
images within the second patch and the reference patch, based 
5 on the correspondent relationships, which were estimated among 
the pixels of the reference patch and the second patch having 
the number of rectangular regions therein that yielded the 
highest degree of correlation; 

obtaining a plurality of intermediate synthesized frames 
10 by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and comparing the degrees of 
correlation for all of the plurality of sampled frames; and 

generating a synthesized frame by synthesizing the 
15 plurality of intermediate synthesized frames. 

3. Amoving image synthesizing apparatus, comprising: 

a sampling means, for sampling two consecutive frames 
from a moving image; 

a correspondent relationship estimating means, for: 
20 providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
same as the reference patch, in the other of the two frames; 
moving and/or deforming the second patch in the other of the 
25 two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
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relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 
reference frame, based on the second patch after the movement 
and/or deformation thereof and the reference patch; 
5 a coordinate converting means, for obtaining a coordinate 

converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 

10 a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 

15 and the correlative value calculating means, to vary the number 
of rectangular regions within the reference patch and the second 
patch in a stepwise manner, to estimate the aforementioned 
correspondent relationships, to obtain the coordinate 
converted frames, and to calculate the correlative values for 

20 each number of rectangular regions, thereby obtaining a 
plurality of correlative values corresponding to the number of 
rectangular regions within the reference patch and the second 
patch; 

a comparing means, for comparing the degrees of 
25 correlation for each number of rectangular regions, based on 
the plurality of the correlative values; and 
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a synthesizing means, for generating a synthesized frame 
having a higher resolution than either of the two frames, by 
administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 
5 correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the highest 
degree of correlation. 

4. A moving image synthesizing apparatus as defined in 
10 claim 3, wherein: 

the correspondent relationships employed in the 
generation of the synthesized frame are those which are 
estimated for each of the rectangular regions. 

5. A moving image synthesizing apparatus as defined in 
15 claim 3, wherein: 

the correspondent relationship estimating means, the 
coordinate converting means, the correlative value calculating 
means, and the synthesizing means employ at least one component 
that constitutes the frames during the estimation of the 
20 correspondent relationships, the obtainment of the coordinate 
converted frames, the calculation of the correlative values, 
and the generation of the synthesized frame. 

6. Amoving image synthesizing apparatus, comprising: 
a sampling means, for sampling at least three consecutive 

25 frames from a moving image; 

a correspondent relationship estimating means, for: 
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providing a reference patch having at least one rectangular 
region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 
5 patch, in another of the plurality of sampled frames; moving 
and/or deforming the second patch in the other frame so that 
an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 

10 the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 
thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 

15 second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 

2 0 frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to vary the number 
of rectangular regions within the reference patch and the second 

25 patch in a stepwise manner, to estimate the aforementioned 
correspondent relationships, to obtain the coordinate 
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converted frames, and to calculate the correlative values for 
each number of rectangular regions, thereby obtaining a 
plurality of correlative values corresponding to the number of 
rectangular regions within the reference patch and the second 
5 patch; 

a comparing means, for comparing the degrees of 
correlation for each number of rectangular regions, based on 
the plurality of the correlative values; and 

a synthesizing means, for: obtaining an intermediate 

10 synthesized frame having a higher resolution than either the 
reference frame of the other frame, by administering 
interpolation calculations on the images within the second 
patch and the reference patch, based on the correspondent 
relationships, which were estimated among the pixels of the 

15 reference patch and the second patch having the number of 
rectangular regions therein that yielded the highest degree of 
correlation; obtaining a plurality of intermediate synthesized 
frames by estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 

20 calculating the plurality of correlative values, and comparing 
the correlations for all of the plurality of sampled frames; 
and generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

7. Amoving image synthesizing apparatus as defined in 

25 claim 6, wherein: 

the correspondent relationships employed in the 
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generation of the intermediate synthesized frame are those 
which are estimated for each of the rectangular regions. 

8. Amoving image synthesizing apparatus as defined in 
claim 6, wherein: 

5 the correspondent relationship estimating means, the 

coordinate converting means, the correlative value calculating 
means, and the synthesizing means employ at least one component 
that constitutes the frames during the estimation of the 
correspondent relationships, the obtainment of the coordinate 
10 converted frames, the calculation of the correlative values, 
the generation of the intermediate synthesized frames, and the 
generation of the synthesized frame. 

9. A program that causes a computer to execute a method 
for synthesizing moving images, comprising the procedures of: 

15 sampling two consecutive frames from a moving image; 

providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
20 reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
25 within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 



161 



patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
5 space of the reference frame, based on the correspondent 
relationships ; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame; 

10 varying the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 

15 thereby obtaining a plurality of correlative values 
corresponding to the number of rectangular regions within the 
reference patch and the second patch; 

comparing the degrees of correlation for each number of 
rectangular regions, based on the plurality of the correlative 

20 values; and 

generating a synthesized frame having a higher resolution 
than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
reference patch, based on the correspondent relationships, 

25 which were estimated among the pixels of the reference patch 
and the second patch having the number of rectangular regions 
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therein that yielded the highest degree of correlation. 

10. A program that causes a computer to execute a method 
for synthesizing moving images, comprising the procedures of: 

sampling at least three consecutive frames from a moving 

5 image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

10 providing a second patch, which is the same as the 

reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
within the reference image; 

15 estimating correspondent relationships among the pixels 

within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

20 obtaining a coordinate converted frame, by coordinate 

converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a correlative value, which represents the 

25 degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame; 
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varying the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
thereby obtaining a plurality of correlative values 
corresponding to the number of rectangular regions within the 
reference patch and the second patch; 

comparing the degrees of correlation for each number of 
rectangular regions, based on the plurality of the correlative 
values; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 
images within the second patch and the reference patch, based 
on the correspondent relationships, which were estimated among 
the pixels of the reference patch and the second patch having 
the number of rectangular regions therein that yielded the 
highest degree of correlation; 

obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and comparing the degrees of 
correlation for all of the plurality of sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 
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11. A method for synthesizing moving images, comprising 
the steps of: 

sampling two consecutive frames from a moving image; 
providing a reference patch having at least one 
5 rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
10 the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
15 patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
20 relationships; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame; 

increasing the number of rectangular regions within the 
25 reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
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obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
until the degree of correlation is greater than or equal to a 
predetermined threshold value; and 
5 generating a synthesized frame having a higher resolution 

than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
reference patch, based on the correspondent relationships, 
which were estimated among the pixels of the reference patch 
10 and the second patch having the number of rectangular regions 
therein that yielded the degree of correlation greater than or 
equal to the threshold value. 

12. A method for synthesizing moving images, comprising 
the steps of: 

15 sampling two consecutive frames from a moving image; 

providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
20 reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
25 within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 



166 



patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
5 space of the reference frame, based on the correspondent 
relationships; 

calculating correlative values, which represent the 
degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame, 

10 for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch, in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 

15 correlative values for each number of rectangular regions, 
until the degree of correlation or a representative degree of 
correlation of all of the rectangular regions, or of the 
rectangular regions at predetermined positions from among all 
of the rectangular regions, other than those for which the 

20 degree of correlation is less than a predetermined value, or 
of a predetermined number of rectangular regions from among all 
of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, 
becomes greater than or equal to a predetermined threshold 

25 value; and 

generating a synthesized frame having a higher resolution 
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than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
reference patch for each region corresponding to the 
rectangular regions, based on the correspondent relationships, 
5 which were estimated among the pixels of the reference patch 
and the second patch having the number of rectangular regions 
therein that yielded the degree of correlation or the 
representative degree of correlation greater than or equal to 
the threshold value. 
10 13. A method for synthesizing moving images, comprising 

the steps of: 

sampling at least three consecutive frames from a moving 

image; 

providing a reference patch having at least one 
15 rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

providing a second patch, which is the same as the 
reference patch, in another of the plurality of sampled frames; 
20 moving and/or deforming the second patch in the other 

frame so that an image within the second patch matches that 
within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
25 the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 



168 



1 



reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
5 relationships; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frames- 
increasing the number of rectangular regions within the 

10 reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
until the degree of correlation is greater than or equal to a 

15 predetermined threshold values- 
obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 
images within the second patch and the reference patch for each 

20 region corresponding to the rectangular regions, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the degree 
of correlation greater than or equal to the threshold value; 

25 obtaining a plurality of intermediate synthesized frames 

by estimating the aforementioned correspondent relationships, 
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obtaining the coordinate converted frames, and calculating the 
plurality of correlative values, for all of the plurality of 
sampled frames; and 

generating a synthesized frame by synthesizing the 
5 plurality of intermediate synthesized frames. 

14. A method for synthesizing moving images, comprising 
the steps of: 

sampling at least three consecutive frames from a moving 

image ; 

10 providing a reference patch having at least one 

rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

providing a second patch, which is the same as the 
15 reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
within the reference image; 

estimating correspondent relationships among the pixels 
20 within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
25 converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
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relationships; 

calculating a correlative value, which represents the 
degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame; 
5 increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 

10 until the degree of correlation or a representative degree of 
correlation of all of the rectangular regions, or of the 
rectangular regions at predetermined positions from among all 
of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, or 

15 of a predetermined number of rectangular regions from among all 
of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, 
becomes greater than or equal to a predetermined threshold 
value; 

20 obtaining an intermediate synthesized frame having a 

higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 
images within the second patch and the reference patch for each 
region corresponding to the rectangular regions, based on the 

25 correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
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number of rectangular regions therein that yielded the degree 
of correlation or the representative degree of correlation 
greater than or equal to the threshold value; 

obtaining a plurality of intermediate synthesized frames 
5 by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
plurality of correlative values, for all of the plurality of 
sampled frames; and 

generating a synthesized frame by synthesizing the 
10 plurality of intermediate synthesized frames. 

15. Amoving image synthesizing apparatus comprising: 

a sampling means, for sampling two consecutive frames 
from a moving image; 

a correspondent relationship estimating means, for: 
15 providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
same as the reference patch, in the other of the two frames; 
moving and/or deforming the second patch in the other of the 
2 0 two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 
reference frame, based on the second patch after the movement 
25 and/or deformation thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 



172 



converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
5 a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means , 

10 and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 

15 values for each number of rectangular regions, until the degree 
of correlation is greater than or equal to a predetermined 
threshold value; and 

a synthesizing means, for generating a synthesized frame 
having a higher resolution than either of the two frames, by 

20 administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the degree 

25 of correlation greater than or equal to the predetermined 
threshold value. 
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16. Amoving image synthesizing apparatus comprising: 

a sampling means, for sampling two consecutive frames 
from a moving image; 

a correspondent relationship estimating means, for: 
5 providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
same as the reference patch, in the other of the two frames; 
moving and/or deforming the second patch in the other of the 
10 two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 
reference frame, based on the second patch after the movement 
15 and/or deformation thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 
20 a correlative value calculating means, for calculating 

a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
25 relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
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number of rectangular regions within the reference patch and 
the second patch in a stepwise manner, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
5 values for each number of rectangular regions, until the degree 
of correlation or a representative degree of correlation of all 
of the rectangular regions, or of the rectangular regions at 
predetermined positions from among all of the rectangular 
regions, other than those for which the degree of correlation 

10 is less than a predetermined value, or of a predetermined number 
of rectangular regions from among all of the rectangular regions, 
other than those for which the degree of correlation is less 
than a predetermined value, becomes greater than or equal to 
a predetermined threshold value; and 

15 a synthesizing means, for generating a synthesized frame 

having a higher resolution than either of the two frames, by 
administering interpolation calculations on the images within 
the second patch and the reference patch for each region 
corresponding to the rectangular regions, based on the 

20 correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the degree 
of correlation or the representative degree of correlation 
greater than or equal to the predetermined threshold value. 

25 17. Amoving image synthesizing apparatus as defined in 

claim 15, wherein: 
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the control means decreases the threshold value, 
according to the increase in the number of rectangular regions, 
until the number of rectangular regions reaches a predetermined 
value, and increases the threshold value, when the number of 
5 rectangular regions exceeds the predetermined value. 

18. Amoving image synthesizing apparatus as defined in 
claim 16, wherein: 

the control means decreases the threshold value, 
according to the increase in the number of rectangular regions, 
10 until the number of rectangular regions reaches a predetermined 
value, and increases the threshold value, when the number of 
rectangular regions exceeds the predetermined value. 

19. Amoving image synthesizing apparatus as defined in 
claim 15, wherein: 

15 the correspondent relationship estimating means, the 

coordinate converting means, the correlative value calculating 
means, and the synthesizing means employ at least one component 
that constitutes the frames during the estimation of the 
correspondent relationships, the obtainment of the coordinate 

20 converted frames, the calculation of the correlative values, 
and the generation of the synthesized frame. 

20. Amoving image synthesizing apparatus as defined in 
claim 16, wherein: 

the correspondent relationship estimating means, the 
25 coordinate converting means, the correlative value calculating 
means, and the synthesizing means employ at least one component 
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that constitutes the frames during the estimation of the 
correspondent relationships, the obtainment of the coordinate 
converted frames, the calculation of the correlative values, 
and the generation of the synthesized frame. 
5 21. Amoving image synthesizing apparatus, comprising: 

a sampling means, for sampling at least three consecutive 
frames from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 

10 region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 
patch, in another of the plurality of sampled frames; moving 
and/or deforming the second patch in the other frame so that 

15 an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 
the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 

20 thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

25 a correlative value calculating means, for calculating 

a correlative value, which represents the degree of correlation 
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between the images within the reference patch of the reference 
frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means , the coordinate converting means, 
5 and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 

10 values for each number of rectangular regions, until the degree 
of correlation is greater than or equal to a predetermined 
threshold value; and 

a synthesizing means, for: obtaining an intermediate 
synthesized frame having a higher resolution than either the 

15 reference frame of the other frame, by: administering 
interpolation calculations on the images within the second 
patch and the reference patch, based on the correspondent 
relationships, which were estimated among the pixels of the 
reference patch and the second patch having the number of 

20 rectangular regions therein that yielded the degree of 
correlation greater than or equal to the threshold value; 
obtaining a plurality of intermediate synthesized frames by 
estimating the aforementioned correspondent relationships 
obtaining the coordinate converted frames, and calculating the 

25 plurality of correlative values, for all of the plurality of 
sampled frames; and generating a synthesized frame by 
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synthesizing the plurality of intermediate synthesized frames . 

22. Amoving image synthesizing apparatus, comprising: 

a sampling means, for sampling at least three consecutive 
frames from a moving image; 
5 a correspondent relationship estimating means, for: 

providing a reference patch having at least one rectangular 
region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 
10 patch, in another of the plurality of sampled frames; moving 
and/or deforming the second patch in the other frame so that 
an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 
15 the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 
thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
20 second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a correlative value, which represents the degree of correlation 
between the images within the reference patch of the reference 
25 frame and the coordinate converted frame; ^ 

a control means, for controlling the correspondent 
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relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner, to estimate the 
5 aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
values for each number of rectangular regions, until the degree 
of correlation or a representative degree of correlation of all 
of the rectangular regions, or of the rectangular regions at 

10 predetermined positions from among all of the rectangular 
regions, other than those for which the degree of correlation 
is less than a predetermined value, or of a predetermined number 
of rectangular regions from among all of the rectangular regions, 
other than those for which the degree of correlation is less 

15 than a predetermined value, becomes greater than or equal to 
a predetermined threshold value; and 

a synthesizing means, for: obtaining an intermediate 
synthesized frame having a higher resolution than either the 
reference frame of the other frame, by: administering 

20 interpolation calculations on the images within the second 
patch and the reference patch for each region corresponding to 
the rectangular regions, based on the correspondent 
relationships, which were estimated among the pixels of the 
reference patch and the second patch having the number of 

25 rectangular regions therein that yielded the degree of 
correlation or the representative degree of correlation greater 
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than or equal to the threshold value; obtaining a plurality of 
intermediate synthesized frames by estimating the 
aforementioned correspondent relationships obtaining the 
coordinate converted frames, and calculating the plurality of 
5 correlative values, for all of the plurality of sampled frames; 
and generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

23. Amoving image synthesizing apparatus as defined in 
claim 21, wherein: 

10 the control means decreases the threshold value, 

according to the increase in the number of rectangular regions, 
until the number of rectangular regions reaches a predetermined 
value, and increases the threshold value, according to the 
increase in the number of rectangular regions, when the number 

15 of rectangular regions exceeds the predetermined value. 

24. Amoving image synthesizing apparatus as defined in 
claim 22, wherein: 

the control means decreases the threshold value, 
according to the increase in the number of rectangular regions, 
20 until the number of rectangular regions reaches a predetermined 
value, and increases the threshold value, according to the 
increase in the number of rectangular regions, when the number 
of rectangular regions exceeds the predetermined value. 

25. Amoving image synthesizing apparatus as defined in 
25 claim 21, wherein: 

the correspondent relationship estimating means, the 
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coordinate converting means, the correlative value calculating 
means, and the synthesizing means employ at least one component 
that constitutes the frames during the estimation of the 
correspondent relationships, the obtainment of the coordinate 
5 converted frames, the calculation of the correlative values, 
the generation of the intermediate synthesized frames, and the 
generation of the synthesized frame. 

26. Amoving image synthesizing apparatus as defined in 
claim 22, wherein: 
10 the correspondent relationship estimating means, the 

coordinate converting means, the correlative value calculating 
means, and the synthesizing means employ at least one component 
that constitutes the frames during the estimation of the 
correspondent relationships, the obtainment of the coordinate 
15 converted frames, the calculation of the correlative values, 
the generation of the intermediate synthesized frames, and the 
generation of the synthesized frame. 

27 . A program that causes a computer to execute a method 
for synthesizing moving images, comprising the procedures of: 
2 0 sampling two consecutive frames from a moving image; 

providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
25 reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
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the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
5 the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
10 space of the reference frame, based on the correspondent 
relationships ; 

calculating a correlative value, which represents 
the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
15 converted frame; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
20 correlative values for each number of rectangular regions, 
until the degree of correlation is greater than or equal to a 
predetermined threshold value; and 

generating a synthesized frame having a higher resolution 
than either of the two frames, by administering interpolation 
25 calculations on the images within the second patch and the 
reference patch, based on the correspondent relationships, 
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which were estimated among the pixels of the reference patch 
and the second patch having the number of rectangular regions 
therein that yielded the degree of correlation greater than or 
equal to the threshold value. 
5 28. A program that causes a computer to execute a method 

for synthesizing moving images, comprising the procedures of: 
sampling two consecutive frames from a moving image; 
providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
10 designated as a reference frame; 

providing a second patch, which is the same as the 
reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
the two frames so that an image within the second patch matches 
15 that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
20 reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

25 calculating correlative values, which represent the 

degree of correlation between the images within the reference 
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patch of the reference frame and the coordinate converted frame, 
for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch, in a stepwise manner, 
5 estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
until the degree of correlation or a representative degree of 
correlation of all of the rectangular regions, or of the 

10 rectangular regions at predetermined positions from among all 
of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, or 
of a predetermined number of rectangular regions from among all 
of the rectangular regions, other than those for which the 

15 degree of correlation is less than a predetermined value, 
becomes greater than or equal to a predetermined threshold 
value; and 

generating a synthesized frame having a higher resolution 
than either of the two frames, by administering interpolation 

20 calculations on the images within the second patch and the 
reference patch for each region corresponding to the 
rectangular regions, based on the correspondent relationships, 
which were estimated among the pixels of the reference patch 
and the second patch having the number of rectangular regions 

25 therein that yielded the degree of correlation or the 
representative degree of correlation greater than or equal to 
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the threshold value. 

29. A program that causes a computer to execute a method 
for synthesizing moving images, comprising the procedures of: 

sampling at least three consecutive frames from a moving 

5 image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

10 providing a second patch, which is the same as the 

reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
within the reference image; 

15 estimating correspondent relationships among the pixels 

within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

20 obtaining a coordinate converted frame, by coordinate 

converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships ; 

calculating a correlative value, which represents the 

25 degree of correlation between the images within the reference 
patch of the reference frame and the coordinate converted frame; 
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increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
5 correlative values for each number of rectangular regions, 
until the degree of correlation is greater than or equal to a 
predetermined threshold value; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
10 frame, by administering interpolation calculations on the 
images within the second patch and the reference patch, based 
on the correspondent relationships, which were estimated among 
the pixels of the reference patch and the second patch having 
the number of rectangular regions therein that yielded the 
15 degree of correlation greater than or equal to the threshold 
valued- 
obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, and calculating the 
20 plurality of correlative values, for all of the plurality of 
sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

30. A program that causes a computer to execute a method 
25 for synthesizing moving images, comprising the procedures of: 
sampling at least three consecutive frames from a moving 
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image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
5 frame; 

providing a second patch, which is the same as the 
reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
10 within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
15 reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships ; 

20 calculating a correlative value, which represents 

the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
converted frame; 

increasing the number of rectangular regions within the 

25 reference patch and the second patch in a stepwise manner, 
estimating the aforementioned correspondent relationships, 
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obtaining the coordinate converted frames, and calculating the 
correlative values for each number of rectangular regions, 
until the degree of correlation or a representative degree of 
correlation of all of the rectangular regions, or of the 
5 rectangular regions at predetermined positions from among all 
of the rectangular regions, other than those for which the 
degree of correlation is less than a predetermined value, or 
of a predetermined number of rectangular regions from among all 
of the rectangular regions, other than those for which the 
10 degree of correlation is less than a predetermined value, 
becomes greater than or equal to a predetermined threshold 
value; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 

15 frame, by administering interpolation calculations on the 
images within the second patch and the reference patch for each 
region corresponding to the rectangular regions, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 

20 number of rectangular regions therein that yielded the degree 
of correlation or the representative degree of correlation 
greater than or equal to the threshold value; 

obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 

25 obtaining the coordinate converted frames, and calculating the 
plurality of correlative values, for all of the plurality of 
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sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

31. A method for synthesizing moving images, comprising 
5 the steps of: 

sampling two consecutive frames from a moving image; 
providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 
10 providing a second patch, which is the same as the 

reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
f the two frames so that an image within the second patch matches 

that within the reference image; 
15 estimating correspondent relationships among the pixels 

within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

20 obtaining a coordinate converted frame, by coordinate 

converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a first correlative value, which represents 

25 the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
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converted frame; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner by 
one step, estimating the aforementioned correspondent 
5 relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
correlative value; 

calculating the variation between the first and second 
correlative values ; 

10 increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 

15 calculating the variation between the first and second 
correlative values of consecutive steps, until the variation 
becomes less than or equal to a predetermined threshold value; 
and 

generating a synthesized frame having a higher resolution 
20 than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
reference patch, based on the correspondent relationships, 
which were estimated among the pixels of either of the reference 
patches and the second patches having the number of rectangular 
25 regions therein that yielded the variation less than or equal 
to the threshold value. 
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32, A method for synthesizing moving images, comprising 
the steps of: 

sampling two consecutive frames from a moving image; 
providing a reference patch having at least one 
5 rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
10 the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
15 patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
20 relationships; 

calculating a first correlative value, which represents 
the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
converted frame, for each of the rectangular regions; 
25 increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner by 
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one step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
correlative value; 
5 calculating the variation between the first and second 

correlative values for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 

10 relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 
calculating the variation between the first and second 
correlative values of consecutive steps, until the variation 
or a representative variation of all of the rectangular regions, 

15 or of the rectangular regions at predetermined positions from 
among all of the rectangular regions, other than those for which 
the variation is greater than or equal to a predetermined value, 
or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 

20 variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value; 
and 

generating a synthesized frame having a higher resolution 
than either of the two frames, by administering interpolation 
25 calculations on the images within the second patch and the 
reference patch for each region corresponding to the 
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rectangular regions, based on the correspondent relationships, 
which were estimated among the pixels of either of the reference 
patches and the second patches having the number of rectangular 
regions therein that yielded the variation or the 
5 representative variation less than or equal to the threshold 
value . 

33. Amethod for synthesizing moving images, comprising 
the steps of: 

sampling at least three consecutive frames from a moving 

10 image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
frame; 

15 providing a second patch, which is the same as the 

reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
within the reference image; 

20 estimating correspondent relationships among the pixels 

within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

25 obtaining a coordinate converted frame, by coordinate 

converting the image within the second patch into a coordinate 
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space of the reference frame, based on the correspondent 

relationships; 

calculating a first correlative value, which represents 

the degree of correlation between the images within the 
5 reference patch of the reference frame and the coordinate 

converted frame ; 

increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner by 

one step, estimating the aforementioned correspondent 
10 relationships, obtaining the coordinate converted frames, and 

calculating the correlative value, thereby obtaining a second 

correlative value; 

calculating the variation between the first and second 

correlative values ; 
15 increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner an 

additional step, estimating the aforementioned correspondent 

relationships, obtaining the coordinate converted frames, 

calculating the first and second correlative values, and 
20 calculating the variation between the first and second 

correlative values of consecutive steps, until the variation 

becomes less than or equal to a predetermined threshold value; 

obtaining an intermediate synthesized frame having a 

higher resolution than either the reference frame of the other 
25 frame, by administering interpolation calculations on the 

images within the second patch and the reference patch, based 
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on the correspondent relationships, which were estimated among 
the pixels of the reference patch and the second patch having 
the number of rectangular regions therein that yielded the 
variation less than or equal to the threshold value; 
5 obtaining a plurality of intermediate synthesized frames 

by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and calculating the variations 
among the plurality of the correlative values, for all of the 
10 plurality of sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 

34. A method for synthesizing moving images, comprising 
the steps of: 

15 sampling at least three consecutive frames from a moving 

image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
2 0 frame; 

providing a second patch, which is the same as the 
reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
frame so that an image within the second patch matches that 
25 within the reference image; 

estimating correspondent relationships among the pixels 
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within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

5 obtaining a coordinate converted frame, by coordinate 

converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

calculating a first correlative value, which represents 
10 the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
converted frame, for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner by 
15 one step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
correlative value; 

calculating the variation between the first and second 
20 correlative values for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
25 calculating the first and second correlative values, and 
calculating the variation between the first and second 
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correlative values of consecutive steps, until the variation 
or a representative variation of all of the rectangular regions, 
or of the rectangular regions at predetermined positions from 
among all of the rectangular regions, other than those for which 
5 the variation is greater than or equal to a predetermined value, 
or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value; 

10 obtaining an intermediate synthesized frame having a 

higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 
images within the second patch and the reference patch for each 
region corresponding to the rectangular regions, based on the 

15 correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the 
variation or the representative variation less than or equal 
to the threshold value; 

20 obtaining a plurality of intermediate synthesized frames 

by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and calculating the variations 
among the plurality of the correlative values, for all of the 

25 plurality of sampled frames; and 

generating a synthesized frame by synthesizing the 
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plurality of intermediate synthesized frames. 

35. Amoving image synthesizing apparatus, comprising: 
a sampling means, for sampling two consecutive frames 
from a moving image; 
5 a correspondent relationship estimating means, for: 

providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
same as the reference patch, in the other of the two frames; 

10 moving and/or deforming the second patch in the other of the 
two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
other frame and the pixels within the reference patch of the 

15 reference frame, based on the second patch after the movement 
and/or deformation thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
second patch into a coordinate space of the reference frame, 

20 based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a first correlative value, which represents the degree of 
correlation between the images within the reference patch of 
the reference frame and the coordinate converted frame; 

25 a control means, for controlling the correspondent 

relationship estimating means, the coordinate converting means, 
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and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
the second patch in a stepwise manner by one step, to estimate 
the aforementioned correspondent relationships, to obtain the 
5 coordinate converted frames, and to calculate the correlative 
value, thereby obtaining a second correlative value; 

a variation calculating means, for calculating the 
variation between the first and second correlative values; 

a comparing means, for controlling the correspondent 

10 relationship estimating means, the coordinate converting means, 
the correlative relationship calculating means, and the 
variation calculating means, to increase the number of 
rectangular regions within the reference patch and the second 
patch in a stepwise manner an additional step, to estimate the 

15 aforementioned correspondent relationships, to obtain the 
coordinate converted frames, to calculate the first and second 
correlative values, and to calculate the variation between the 
first and second correlative values of consecutive steps, until 
the variation becomes less than or equal to a predetermined 

20 threshold value; and 

a synthesizing means, for generating a synthesized frame 
having a higher resolution than either of the two frames, by 
administering interpolation calculations on the images within 
the second patch and the reference patch, based on the 

25 correspondent relationships, which were estimated among the 
pixels of either of the reference patches and the second patches 
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having the number of rectangular regions therein that yielded 
the variation less than or equal to the threshold value. 

36. Amoving image synthesizing apparatus, comprising: 

a sampling means, for sampling two consecutive frames 
5 from a moving image; 

a correspondent relationship estimating means, for: 
providing a reference patch having at least one rectangular 
region therein, in one of the two frames, which is designated 
as a reference frame; providing a second patch, which is the 
10 same as the reference patch, in the other of the two frames; 
moving and/or deforming the second patch in the other of the 
two frames so that an image within the second patch matches that 
within the reference image; and estimating correspondent 
relationships among the pixels within the second patch of the 
15 other frame and the pixels within the reference patch of the 
reference frame, based on the second patch after the movement 
and/or deformation thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 
20 second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a first correlative value, which represents the degree of 
correlation between the images within the reference patch of 
25 the reference frame and the coordinate converted frame, for each 
of the rectangular regions; 
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a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 
5 the second patch in a stepwise manner by one step, to estimate 
the aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
value, thereby obtaining a second correlative value; 

a variation calculating means, for calculating the 

10 variation between the first and second correlative values for 
each of the rectangular regions; 

a comparing means, for controlling the correspondent 
relationship estimating means , the coordinate converting means, 
the correlative relationship calculating means, and the 

15 variation calculating means, to increase the number of 
rectangular regions within the reference patch and the second 
patch in a stepwise manner an additional step, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, to calculate the first and second 

20 correlative values, and to calculate the variation between the 
first and second correlative values of consecutive steps, until 
the variation or a representative variation of all of the 
rectangular regions, or of the rectangular regions at 
predetermined positions from among all of the rectangular 

25 regions, other than those for which the variation is greater 
than or equal to a predetermined value, or of a predetermined 
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number of rectangular regions from among all of the rectangular 
regions, other than those for which the variation is greater 
than or equal to a predetermined value, becomes less than or 
equal to a predetermined threshold value; and 
5 a synthesizing means, for generating a synthesized frame 

having a higher resolution than either of the two frames, by 
administering interpolation calculations on the images within 
the second patch and the reference patch for each region 
corresponding to the rectangular regions, based on the 

10 correspondent relationships, which were estimated among the 
pixels of either of the reference patches and the second patches 
having the number of rectangular regions therein that yielded 
the variation or the representative variation less than or equal 
to the threshold value. 

15 37. Amoving image synthesizing apparatus as defined in 

claim 35, wherein: 

the control means sets the predetermined threshold value 
to be larger, according to the increase in the number of 
rectangular regions, when the number of rectangular regions 

20 exceeds a predetermined value. 

38. Amoving image synthesizing apparatus as defined in 
claim 36, wherein: 

the control means sets the predetermined threshold value 
to be larger, according to the increase in the number of 

25 rectangular regions, when the number of rectangular regions 
exceeds a predetermined value. 
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39. Amoving image synthesizing apparatus as defined in 
claim 35, wherein: 

the correspondent relationship estimating means, the 
coordinate converting means, the correlative value calculating 
5 means, the control means, the variation calculating means, the 
comparing means, and the synthesizing means employ at least one 
component that constitutes the frames during the estimation of 
the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
10 values, the calculation of the variations, and the generation 
of the synthesized frame. 

40. Amoving image synthesizing apparatus as defined in 
claim 3 6, wherein : 

the correspondent relationship estimating means, the 
15 coordinate converting means, the correlative value calculating 
means, the control means, the variation calculating means, the 
comparing means, and the synthesizing means employ at least one 
component that constitutes the frames during the estimation of 
the correspondent relationships, the obtainment of the 
20 coordinate converted frames, the calculation of the correlative 
values, the calculation of the variations, and the generation 
of the synthesized frame. 

41. Amoving image synthesizing apparatus, comprising: 
a sampling means, for sampling at least three consecutive 

2 5 frames from a moving image; 

a correspondent relationship estimating means, for: 
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providing a reference patch having at least one rectangular 
region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 
5 patch, in another of the plurality of sampled frames; moving 
and/or deforming the second patch in the other frame so that 
an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 

10 the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 
thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 

15 second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 

a correlative value calculating means, for calculating 
a first correlative value, which represents the degree of 
correlation between the images within the reference patch of 

20 the reference frame and the coordinate converted frame; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 

25 the second patch in a stepwise manner by one step, to estimate 
the aforementioned correspondent relationships, to obtain the 
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coordinate converted frames, and. to calculate the correlative 
value, thereby obtaining a second correlative value; 

a variation calculating means, for calculating the 
variation between the first and second correlative values; 
5 a comparing means, for controlling the correspondent 

relationship estimating means, the coordinate converting means, 
the correlative value calculating means, and the variation 
calculating means, to increase the number of rectangular 
regions within the reference patch and the second patch in a 

10 stepwise manner an additional step, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, to calculate the first and second 
correlative values, and to calculate the variation between the 
first and second correlative values of consecutive steps, until 

15 the variation becomes less than or equal to a predetermined 
threshold value; and 

a synthesizing means, for: obtaining an intermediate 
synthesized frame having a higher resolution than either the 
reference frame of the other frame, by administering 

20 interpolation calculations on the images within the second 
patch and the reference patch, based on the correspondent 
relationships, which were estimated among the pixels of the 
reference patch and the second patch having the number of 
rectangular regions therein that yielded the variation less 

25 than or equal to the threshold value; obtaining a plurality of 
intermediate synthesized frames by estimating the 
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aforementioned correspondent relationships, obtaining the 
coordinate converted frames, calculating the plurality of 
correlative values, and calculating the variations among the 
plurality of the correlative values, for all of the plurality 
5 of sampled frames; and generating a synthesized frame by 
synthesizing the plurality of intermediate synthesized frames. 
42 • Amoving image synthesizing apparatus, comprising: 
a sampling means, for sampling at least three consecutive 
frames from a moving image; 

10 a correspondent relationship estimating means, for: 

providing a reference patch having at least one rectangular 
region therein, in a first frame from among the at least three 
sampled frames, which is designated as a reference frame; 
providing a second patch, which is the same as the reference 

15 patch, in another of the plurality of sampled frames; moving, 
and/or deforming the second patch in the other frame so that 
an image within the second patch matches that within the 
reference image; and estimating correspondent relationships 
among the pixels within the second patch of the other frame and 

20 the pixels within the reference patch of the reference frame, 
based on the second patch after the movement and/or deformation 
thereof and the reference patch; 

a coordinate converting means, for obtaining a coordinate 
converted frame, by coordinate converting the image within the 

25 second patch into a coordinate space of the reference frame, 
based on the correspondent relationships; 
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a correlative value calculating means, for calculating 
a first correlative value, which represents the degree of 
correlation between the images within the reference patch of 
the reference frame and the coordinate converted frame, for each 
5 of the rectangular regions; 

a control means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
and the correlative value calculating means, to increase the 
number of rectangular regions within the reference patch and 

10 the second patch in a stepwise manner by one step, to estimate 
the aforementioned correspondent relationships, to obtain the 
coordinate converted frames, and to calculate the correlative 
value, thereby obtaining a second correlative value; 

a variation calculating means, for calculating the 

15 variation between the first and second correlative values for 
each of the rectangular regions; 

a comparing means, for controlling the correspondent 
relationship estimating means, the coordinate converting means, 
the correlative value calculating means, and the variation 

20 calculating means, to increase the number of rectangular 
regions within the reference patch and the second patch in a 
stepwise manner an additional step, to estimate the 
aforementioned correspondent relationships, to obtain the 
coordinate converted frames, to calculate the first and second 

25 correlative values, and to calculate the variation between the 
first and second correlative values of consecutive steps, until 
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the variation of all of the rectangular regions, or of the 
rectangular regions at predetermined positions from among all 
of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 
5 or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 
variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value; 
and 

10 a synthesizing means, for: obtaining an intermediate 

synthesized frame having a higher resolution than either the 
reference frame of the other frame, by administering 
interpolation calculations on the images within the second 
patch and the reference patch, based on the correspondent 

15 relationships, which were estimated among the pixels of the 
reference patch and the second patch having the number of 
rectangular regions therein that yielded the variation or the 
representative variation less than or equal to the threshold 
value; obtaining a plurality of intermediate synthesized frames 

20 by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and calculating the variations 
among the plurality of the correlative values, for all of the 
plurality of sampled frames; and generating a synthesized frame 

25 by synthesizing the plurality of intermediate synthesized 
frames . 
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43. Amoving image synthesizing apparatus as defined in 
claim 41, wherein: 

the control means sets the predetermined threshold value 
to be larger, according to the increase in the number of 
5 rectangular regions, when the number of rectangular regions 
exceeds a predetermined value. 

44. Amoving image synthesizing apparatus as defined in 
claim 42, wherein: 

the control means sets the predetermined threshold value 
10 to be larger, according to the increase in the number of 
rectangular regions, when the number of rectangular regions 
exceeds a predetermined value. 

45. Amoving image synthesizing apparatus as defined in 
claim 41, wherein: 

15 the correspondent relationship estimating means, the 

coordinate converting means, the correlative value calculating 
means, the control means, the variation calculating means, the 
comparing means, and the synthesizing means employ at least one 
component that constitutes the frames during the estimation of 

20 the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
values, the calculation of the variations, the generation of 
the intermediate synthesized frames, and the generation of the 
synthesized frame . 

25 46. Amoving image synthesizing apparatus as defined in 

claim 42, wherein: 
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the correspondent relationship estimating means, the 
coordinate converting means, the correlative value calculating 
means, the control means, the variation calculating means, the 
comparing means, and the synthesizing means employ at least one 
5 component that constitutes the frames during the estimation of 
the correspondent relationships, the obtainment of the 
coordinate converted frames, the calculation of the correlative 
values, the calculation of the variations, the generation of 
the intermediate synthesized frames, and the generation of the 
10 synthesized frame. 

47. A program that causes a computer to execute a method 
for synthesizing moving images, comprising the procedures of: 

sampling two consecutive frames from a moving image; 

providing a reference patch having at least one 
15 rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
20 the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
25 patch after the movement and/or deformation thereof and the 
reference patch; 
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obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships; 

5 calculating a first correlative value, which represents 

the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
converted frame; 

increasing the number of rectangular regions within the 

10 reference patch and the second patch in a stepwise manner by 
one step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
correlative value; 

15 calculating the variation between the first and second 

correlative values; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 

20 relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 
calculating the variation between the first and second 
correlative values of consecutive steps, until the variation 
becomes less than or equal to a predetermined threshold value; 

25 and 

generating a synthesized frame having a higher resolution 
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than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
reference patch, based on the correspondent relationships, 
which were estimated among the pixels of either of the reference 
5 patches and the second patches having the number of rectangular 
regions therein that yielded the variation less than or equal 
to the threshold value. 

48. A program that causes a computer to execute a method 
for synthesizing moving images, comprising the procedures of: 
10 sampling two consecutive frames from a moving image; 

providing a reference patch having at least one 
rectangular region therein, in one of the two frames, which is 
designated as a reference frame; 

providing a second patch, which is the same as the 
15 reference patch, in the other of the two frames; 

moving and/or deforming the second patch in the other of 
the two frames so that an image within the second patch matches 
that within the reference image; 

estimating correspondent relationships among the pixels 
20 within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
25 converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 



213 



relationships; 

calculating a first correlative value, which represents 
the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
5 converted frame, for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner by 
one step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 

10 calculating the correlative value, thereby obtaining a second 
correlative valuer- 
calculating the variation between the first and second 
correlative values for each of the rectangular regions; 

increasing the number of rectangular regions within the 

15 reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 
calculating the variation between the first and second 

20 correlative values of consecutive steps, until the variation 
or a representative variation of all of the rectangular regions, 
or of the rectangular regions at predetermined positions from 
among all of the rectangular regions, other than those for which 
the variation is greater than or equal to a predetermined value, 

25 or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 
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variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value; 
and 

generating a synthesized frame having a higher resolution 
5 than either of the two frames, by administering interpolation 
calculations on the images within the second patch and the 
reference patch for each region corresponding to the 
rectangular regions, based on the correspondent relationships, 
which were estimated among the pixels of either of the reference 
10 patches and the second patches having the number of rectangular 
regions therein that yielded the variation or the 
representative variation less than or equal to the threshold 
value. 

49. A program that causes a computer to execute a method 
15 for synthesizing moving images, comprising the procedures of: 
sampling at least three consecutive frames from a moving 

image; 

providing a reference patch having at least one 
rectangular region therein, in a first frame from among the at 
20 least three sampled frames, which is designated as a reference 
frame; 

providing a second patch, which is the same as the 
reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
25 frame so that an image within the second patch matches that 
within the reference image; z 
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estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
patch after the movement and/or deformation thereof and the 
5 reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
relationships ; 

10 calculating a first correlative value, which represents 

the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
converted frame; 

increasing the number of rectangular regions within the 

15 reference patch and the second patch in a stepwise manner by 
one step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
correlative value; 

20 calculating the variation between the first and second 

correlative values; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 

25 relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 
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calculating the variation between the first and second 
correlative values of consecutive steps, until the variation 
becomes less than or equal to a predetermined threshold value; 

obtaining an intermediate synthesized frame having a 
5 higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 
images within the second patch and the reference patch, based 
on the correspondent relationships, which were estimated among 
the pixels of the reference patch and the second patch having 
10 the number of rectangular regions therein that yielded the 
variation less than or equal to the threshold value; 

obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
15 plurality of correlative values, and calculating the variations 
among the plurality of the correlative values, for all of the 
plurality of sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames. 
20 50. A program that causes a computer to execute a method 

for synthesizing moving images, comprising the procedures of: 

sampling at least three consecutive frames from a moving 

image; 

providing a reference patch having at least one 
25 rectangular region therein, in a first frame from among the at 
least three sampled frames, which is designated as a reference 
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frame; 

providing a second patch, which is the same as the 
reference patch, in another of the plurality of sampled frames; 

moving and/or deforming the second patch in the other 
5 frame so that an image within the second patch matches that 
within the reference image; 

estimating correspondent relationships among the pixels 
within the second patch of the other frame and the pixels within 
the reference patch of the reference frame, based on the second 
10 patch after the movement and/or deformation thereof and the 
reference patch; 

obtaining a coordinate converted frame, by coordinate 
converting the image within the second patch into a coordinate 
space of the reference frame, based on the correspondent 
15 relationships; 

calculating a first correlative value, which represents 
the degree of correlation between the images within the 
reference patch of the reference frame and the coordinate 
converted frame, for each of the rectangular regions; 
20 increasing the number of rectangular regions within the 

reference patch and the second patch in a stepwise manner by 
one step, estimating the aforementioned correspondent 
relationships, obtaining the coordinate converted frames, and 
calculating the correlative value, thereby obtaining a second 
25 correlative value; 

calculating the variation between the first and second 
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correlative values for each of the rectangular regions; 

increasing the number of rectangular regions within the 
reference patch and the second patch in a stepwise manner an 
additional step, estimating the aforementioned correspondent 
5 relationships, obtaining the coordinate converted frames, 
calculating the first and second correlative values, and 
calculating the variation between the first and second 
correlative values of consecutive steps, until the variation 
or a representative variation of all of the rectangular regions, 

10 or of the rectangular regions at predetermined positions from 
among all of the rectangular regions, other than those for which 
the variation is greater than or equal to a predetermined value, 
or of a predetermined number of rectangular regions from among 
all of the rectangular regions, other than those for which the 

15 variation is greater than or equal to a predetermined value, 
becomes less than or equal to a predetermined threshold value; 

obtaining an intermediate synthesized frame having a 
higher resolution than either the reference frame of the other 
frame, by administering interpolation calculations on the 

20 images within the second patch and the reference patch for each 
region corresponding to the rectangular regions, based on the 
correspondent relationships, which were estimated among the 
pixels of the reference patch and the second patch having the 
number of rectangular regions therein that yielded the 

25 variation or the representative variation less than or equal 
to the threshold value; 
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obtaining a plurality of intermediate synthesized frames 
by estimating the aforementioned correspondent relationships, 
obtaining the coordinate converted frames, calculating the 
plurality of correlative values, and calculating the variations 
5 among the plurality of the correlative values, for all of the 
plurality of sampled frames; and 

generating a synthesized frame by synthesizing the 
plurality of intermediate synthesized frames: 

51. A computer readable medium having recorded therein 
10 the program defined in claim 9. 

52. A computer readable medium having recorded therein 
the program defined in claim 10. 

53. A computer readable medium having recorded therein 
the program defined in claim 27. 
15 54 . A computer readable medium having recorded therein 

the program defined in claim 28. 

55. A computer readable medium having recorded therein 
the program defined in claim 29. 

56. A computer readable medium having recorded therein 
20 the program defined in claim 30. 

57. A computer readable medium having recorded therein 
the program defined in claim 47. 

58. A computer readable medium having recorded therein 
the program defined in claim 48. 

25 59. A computer readable medium having recorded therein 

the program defined in claim 49. 
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60. A computer readable medium having recorded therein 
the program defined in claim 50, 
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